Skip to content

feat(cluster): move etcd's peer traffic onto the servers' LAN addresses - #257

Merged
Mearman merged 8 commits into
mainfrom
feat/etcd-peers-over-lan
Nov 16, 2026
Merged

Mearman merged 8 commits into
mainfrom
feat/etcd-peers-over-lan

Conversation

@Mearman

@Mearman Mearman commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Refs #256

When a server's mesh device stops working (an expired node key, for example), its etcd member loses its peers even if every server is on the same LAN, because etcd's peer and client URLs are mesh addresses. This adds an opt-in path that puts them on the servers' LAN addresses.

github_runner_cluster_etcd_lan_address (per server, explicit rather than detected, since a host can have several LAN addresses and the one that matters is the one the others reach) and github_runner_cluster_etcd_peers_over_lan (per group). The move goes through k3s's --node-ip, which with --vpn-auth changes only etcd's address: the node IP, flannel and the API stay on the mesh. No --etcd-arg is used, so the temporary etcd k3s starts on every restart is untouched (the known issue in docs/non-obvious-behaviour.md).

k3s registers one peer URL per member and etcd refuses a join whose member URLs do not match, so a member cannot carry both addresses. The fallback to the mesh is therefore decided per run: a member stays on the mesh while another server cannot reach its LAN address, or another server's peer certificate does not yet name its own LAN address.

Limitation: it needs each server to be a Docker node on the host network of a Docker host that has the LAN address itself (a bridged VM on macOS). On the bridge network the VM's NAT rewrites the peer's source address, which fails etcd's peer certificate check, so the role refuses that, and also refuses native nodes and headscale_in_cluster. All the checks run before anything changes, --check included.

Rollout, all in the role README's "etcd peers over the LAN" section:

  1. Set the address on every server and eligible host and run. Each server is recreated in its turn (the existing quorum guard and Ready wait) with the address added to its certificates. No member moves.
  2. Set github_runner_cluster_etcd_peers_over_lan: true and run once per server. Each run moves at most one member. Before moving it checks that every server is Ready, every member is healthy, there have been no read-index timeouts in any server's log for the last 10 minutes, every other server reaches the address and every other certificate names its own address. Then it runs etcdctl member update, recreates the server on the new address, and checks Ready, membership, health and a 60 s quiet log window afterwards. If anything fails, the member is updated back, .env restored and the server restarted on its old address, and the run fails.
  3. To move back, set it to false and run once per server. Removing an address while .env still starts etcd on it is refused.

A host that sets neither gets the same k3s arguments, .env and Compose files as before (tested exactly). The role controller's probe now dials the etcd.k3s.cattle.io/node-address annotation instead of the InternalIP, so its peer reading stays right after a move.

Nothing was applied to any live cluster. To opt in, the operator sets the addresses in the fleet's host_vars and follows the steps above, one run per stage.

Tested: the new unit modules (test_cluster_etcd_lan.py and test_cluster_etcd_lan_role.py, the second with real certificates read by openssl), the full unit suite in CI, plus shellcheck, yamllint, the playbook syntax check and ansible-lint (production). I made 15 deliberate breakages to the entrypoint, env.j2, the Compose file list, the filters, the decision tasks and the probe, and the tests caught every one. The new etcd_lan mesh scenario passes in CI. It runs three servers on the host network of dind daemons that share one network, cuts one server's Tailscale interface and sees its member drop out of etcd, then moves every member onto the LAN one run at a time while writing through a server that is not moving. A rerun restarts nothing. The same cut then leaves every member healthy and the cut server writable, and the scenario finishes by moving the members back.

What no test here covers is a real Colima bridged VM on macOS and a real expired node key.

role_swap failed once on this branch with "voter grtest-native is connected to 3 of its 2 peers", the same refusal seen on another branch before this change. It comes from etcd's metrics, not the address the probe dials, and is tracked in #259. I also filed #258 while reading k3s's source: the docs' 100/1000 ms default etcd timing is 500/5000 ms in v1.34.1.

The probe timed a TCP connect to every voter's etcd peer port at the
node's InternalIP, which is where etcd listens only while its member
uses the mesh address. It now dials the address k3s keeps in the
etcd.k3s.cattle.io/node-address annotation, the one the member's peer
URL is built from, so the reading stays right for a member that has
moved onto its LAN address.
A server whose mesh device stops working loses its etcd peers with it,
even when every server shares a LAN, because etcd's peer and client
URLs are mesh addresses. github_runner_cluster_etcd_lan_address (per
server) and github_runner_cluster_etcd_peers_over_lan (per group) put
them on the LAN instead, through k3s's --node-ip, which with
--vpn-auth moves only etcd: the node IP, flannel and the API stay on
the mesh. No --etcd-arg is used, so k3s's temporary etcd is untouched.

k3s registers one peer URL per member and etcd refuses a join whose
member URLs differ, so a member has one address and the fallback to
the mesh is decided per run: a member stays on the mesh while another
server does not reach its LAN address or lacks its own LAN address in
its peer certificate. Setting the address only adds it to the
certificates. With the switch on, each run moves at most one member
(etcdctl member update, then a recreate onto the new address) after
checking every server Ready, every member healthy and no read-index
timeouts in the servers' logs, and checks the same afterwards; a move
that fails is updated back, .env restored and the server restarted on
its old address. Turning the switch off moves members back one per
run. The address must be on a Docker node on the host network: on the
bridge network the VM's NAT rewrites the peer's source address, which
fails etcd's certificate check, so the role refuses it, and refuses
native nodes and headscale_in_cluster.

A host that sets neither keeps the same k3s arguments, .env and
Compose files as before.
… mesh

The etcd_lan scenario runs three servers on the host network of Docker
daemons sharing the test network. It first shows that cutting one
server's Tailscale interface takes its member out of etcd while etcd
is on the mesh, then sets the LAN addresses (no member moves), turns
github_runner_cluster_etcd_peers_over_lan on and expects one member on
the LAN per run with writes succeeding throughout and a rerun
restarting nothing, repeats the cut and expects every member healthy
and the cut server writable, and turns the switch off to move them
back one per run.
etcdctl endpoint health exits non-zero whenever a member is unhealthy,
so under pipefail the scenario's check took the case it looks for as
not yet. Its output is now read whatever its status.
…sses

The check refused real addresses in the mesh test. It now parses the
Docker host's addresses as the host network's own LAN check does and
names the addresses it found when it refuses.
grep -q stops reading at its first match, so the echo feeding it fails
with a broken pipe and pipefail fails the check that the report was made.
Each run moves one server's member and recreates its container, which
takes its own API server down; writes through it measured that, not
etcd's quorum. Each run now writes through the next server instead.
The etcd_lan scenario starts a writer per move, and each began at the
same names, so the second one's writes failed as already existing.
@Mearman
Mearman marked this pull request as ready for review November 16, 2026 14:09
@Mearman
Mearman merged commit d7dface into main Nov 16, 2026
58 of 60 checks passed
@Mearman
Mearman deleted the feat/etcd-peers-over-lan branch November 16, 2026 14:12
@github-actions

Copy link
Copy Markdown

🎉 This PR is included in version 3.3.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant